Method and apparatus for generating depth image

ABSTRACT

A method of generating a depth image includes irradiating an object with a light which is generated from a light source, acquiring a plurality of phase difference signals which have different phase differences from one another, by sensing a reflection light reflected from the object, generating a first depth image based on the plurality of phase difference signals, generating a second depth image based on phase difference signals in which a motion artifact has not occurred, among the plurality of phase difference signals, generating a third depth image by combining the first depth image and the second depth image.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of Korean Patent Application No.10-2014-0059959, filed on May 19, 2014, in the Korean IntellectualProperty Office, the disclosure of which is incorporated herein in itsentirety by reference.

BACKGROUND

1. Field

Example embodiments relate to a method and apparatus for generating adepth image, and more particularly, to a method and apparatus forgenerating a depth image which provide an improved depth image byremoving a motion artifact and an offset.

2. Description of the Related Art

Recently, technologies such as 3D cameras, etc. for capturing distanceinformation of an object are being developed. One such technology uses atime of flight (TOF) method which measures a distance between acapturing apparatus and an object by measuring a turnaround time oflight.

According to the TOF method, light of a specific wavelength, forexample, a near infrared ray of 850 nm, is projected onto an object byusing a light-emitting diode (LED) or a laser diode (LD) and the lighthaving the same wavelength is reflected from the object and is measuredor captured by a photodiode or a camera and light processing isperformed for extracting a depth image. A variety of TOF methods forperforming the light processing which includes a series of operationssuch as light source projection, reflection from an object, opticalmodulation, capturing, and processing, have been developed.

A depth image using a TOF camera is acquired by calculating a phasedifference between an irradiation light signal and a reflection lightsignal corresponding to the irradiation light reflected from an objectduring an integration time per each frame. In a case in which the TOFcamera or the object oscillates during a time shorter than theintegration time, there may be a variation in a phase of the reflectionlight signal. In this case, the acquired depth information according thereflection light signal may include incorrect values, thus causing amotion artifact to occur.

SUMMARY

Provided are a method and an apparatus for generating a depth imagegenerating a depth image from which a motion artifact and an offset havebeen removed.

Additional aspects will be set forth in part in the description whichfollows and, in part, will be apparent from the description, or may belearned by practice of the presented example embodiments.

According to at least one example embodiment, a method of generating adepth image includes irradiating an object with a light which isgenerated from a light source, acquiring a plurality of phase differencesignals which have different phase differences from one another, bysensing a reflection light reflected from the object, generating a firstdepth image based on the plurality of phase difference signals,generating a second depth image based on phase difference signals inwhich a motion artifact has not occurred, among the plurality of phasedifference signals, generating a third depth image by combining thefirst depth image and the second depth image.

The acquiring of the plurality of phase difference signals may includeacquiring a plurality of phase difference signals based on controlsignals having different phase differences from one another.

The generating of the first depth image may include generating firstphase difference images based on the plurality of phase differencesignals, generating a first depth image based on the first phasedifference images, and the generating of the second depth image mayinclude generating second phase difference images based on phasedifference signals in which a motion artifact has not occurred, fromamong the plurality of phase difference signals, and generating thesecond depth image based on the second phase difference images.

The second depth image may be a depth image from which a motion artifacthas been removed, and an offset effect may be included in the seconddepth image.

The generating of the third depth image may include generating a fourthdepth image by changing depth values of the second depth image intocorresponding depth values of the first depth image, and generating thethird depth image by combining the first depth image and the fourthdepth image.

The example method may further include calculating a weight factor withrespect to the first depth image and the fourth depth image, based on anamplitude of the motion artifact, wherein the generating of the thirddepth image by combining the first depth image and the fourth depthimage may include combining the first depth image and the fourth depthimage based on the calculated weight factor.

The generating of the third depth image may include acquiring a firstwavelet coefficient by performing wavelet transform on the first depthimage, acquiring a second wavelet coefficient by performing wavelettransform on the fourth depth image, acquiring a third waveletcoefficient based on the first wavelet coefficient and the secondwavelet coefficient, and acquiring a third depth image by performinginverse-wavelet transform on the third wavelet coefficient.

The third depth image may include depth information with respect to theobject shown in the first depth image, and shape information withrespect to the object shown in the second depth image.

The third depth image may include an image from which a motion artifactshown in the first depth image and an offset effect shown in the seconddepth image have been removed.

According to at least one example embodiment, an apparatus forgenerating a depth image includes a light irradiator irradiating anobject with a light generated from a light source, a sensor acquiring aplurality of phase difference signals which have different phasedifferences from one another by sensing a reflection light reflectedfrom the object, and an image processor generating a first depth imagebased on the plurality of phase difference signals, generating a seconddepth image based on phase difference signals in which a motion artifacthas not occurred, from among the plurality of phase difference signals,and generating a third depth image by combining the first depth imageand the second depth image.

The sensor may acquire the plurality of phase difference signals basedon control signals having different phase differences from one another.

The image processor may generate first phase difference images based onthe plurality of phase difference signals, generate the first depthimage based on the first phase difference images, generate second phasedifference images based on phase difference signals in which a motionartifact has not occurred, from among the plurality of phase differencesignals, and generate the second depth image based on the second phasedifference images.

The second depth image may include a depth image from which a motionartifact has been removed, and an offset effect may be included in thesecond depth image.

The image processor may generate a fourth depth image by changing depthvalues of the second depth image into corresponding depth values of thefirst depth image, and generate a third depth image by combining thefirst depth image and the fourth depth image.

The image processor may calculate a weight factor with respect to thefirst depth image and the fourth depth image based on an amplitude ofthe motion artifact, and combine the first depth image and the fourthdepth image based on the calculated weight factor.

The image processor may acquire a first wavelet coefficient byperforming wavelet transform on the first depth image, acquire a secondwavelet coefficient by performing wavelet transform on the fourth depthimage, acquire a third wavelet coefficient based on the first waveletcoefficient and the second wavelet coefficient, and acquire the thirddepth image by performing inverse-wavelet transform on the third waveletcoefficient.

The third depth image may include depth information with respect to theobject shown in the first depth image, and shape information withrespect to the object shown in the second depth image.

The third depth image may include a depth image from which a motionartifact shown in the first depth image and an offset effect shown inthe second depth image have been removed.

BRIEF DESCRIPTION OF THE DRAWINGS

These and/or other example embodiments will become apparent and morereadily appreciated from the following description of the exampleembodiments, taken in conjunction with the accompanying drawings inwhich:

FIG. 1 is a block diagram showing a depth image generating apparatusaccording to example embodiments;

FIGS. 2A, 2B, 2C, 2D, 2E, 2F are diagrams explaining an irradiationlight signal, a reflection light signal, and phase difference signalsaccording to example embodiments;

FIG. 3A shows phase difference images according to example embodiments,and FIG. 3B shows a depth image according to example embodiments;

FIG. 4A shows a depth image from which the motion artifact has not beenremoved and FIG. 4B shows a depth image from which the motion artifacthas been removed, and FIG. 4C shows a depth image in which a matchinghas been performed, according to example embodiments;

FIG. 5 is a graph showing depth values of a depth image from which amotion artifact has not been removed and depth values of another depthimage from which the motion artifact has been removed, according toexample embodiments;

FIG. 6 shows a depth image from which a motion artifact and an offseteffect have been removed, according to example embodiments;

FIG. 7 is a flowchart illustrating a method of generating a depth imageaccording to example embodiments; and

FIG. 8 is a flowchart illustrating a method of generating a depth imageaccording to example embodiments.

DETAILED DESCRIPTION

The terms used in this specification are those general terms currentlywidely used in the art in consideration of functions in regard toexample embodiments, but the terms may vary according to the intentionof those of ordinary skill in the art, precedents, or new technology inthe art. Also, specified terms may be selected by the applicant, and inthis case, the detailed meaning thereof will be described in thedetailed description of the invention. Thus, the terms used in thespecification should be understood not as simple names but based on themeaning of the terms and the overall description of the invention.

Throughout the specification, it will also be understood that when acomponent “includes” an element, unless there is another oppositedescription thereto, it should be understood that the component does notexclude another element but may further include another element. Inaddition, terms such as “ . . . unit”, “ . . . module”, or the likerefer to units that perform at least one function or operation, and theunits may be implemented as hardware or software or as a combination ofhardware and software.

Reference will now be made in detail to example embodiments, examples ofwhich are illustrated in the accompanying drawings, wherein likereference numerals refer to like elements throughout. In this regard,the present embodiments may have different forms and should not beconstrued as being limited to the descriptions set forth herein.Accordingly, example embodiments are merely described below, byreferring to the figures, to explain aspects of the present description.

FIG. 1 is a block diagram showing a depth image generating apparatus 100according to example embodiments.

Referring to FIG. 1, the image generating apparatus 100 may include alight irradiator 110, a controller 120, a sensor 130, a motion artifactdetermining unit 140, and an image processor 150.

The light irradiator 110 may include a light source for generating alight having a predetermined wavelength, and a light source driver fordriving the light source. The light source may be a light emitting diode(LED) or a laser diode (LD), which emits light in the form of a nearinfrared (NIR) ray having a wavelength of about 850 nm, which isinvisible to the naked eye. However, the light source and the wavelengthof the light emitted by the light source are not limited thereto, and avariety of light sources and wavelengths and may be used.

The light source driver may drive the light source according to acontrol signal received from the controller 120. An irradiation lightsignal which is emitted from the light source and used to irradiate anobject 50 may have a continuous periodic function form having apredetermined period. For example, the irradiation light signal may havea specially defined waveform such as a sine wave, a ramp wave, arectangular wave, etc., and optionally, the irradiation light signal mayhave an undefined waveform.

The light irradiator 110 irradiates the object with an irradiation lightsignal and the irradiation light signal is reflected and returned fromthe object 50 as a reflection signal which is sensed by the sensor 130.The sensor 130 may include a photonic sensing device such as a pinnedphoto diode (PPD), a charge-coupled device (CCD) image sensor, etc.

The sensor 130 receives the reflection light signal, generateselectrons, transmits the generated electrons to an integrator whichintegrates the electrons, and measures a quantity of the integratedelectrons (a charge quantity). An integrating time and an integratingperiod of the electrons may be defined in advance.

The controller 120 may generate a control signal which controls a timingfor integrating electrons which are generated by the sensor 130 when thesensor 130 receives the reflection light signal. The sensor 130 mayinclude a plurality of integrators, and may transmit the generatedelectrons to the plurality of integrators according to the controlsignal.

The depth image generating apparatus using the time of flight (TOF)method generates L phases (where L is an integer) which are differentfrom one another, and in a case in which the depth image generatingapparatus includes M integrators (a storage for charges), the depthimage generating apparatus may be operated by an L-phase/M-tap method.

Accordingly, the sensor 130 may sense the reflection light signal andintegrate a plurality of charge quantities, and acquire phase differencesignals corresponding to the plurality of charge quantities integrated,according to the control signal which haves phases different from oneanother. A detailed description thereof will be given below, byreferencing FIG. 2.

The motion artifact determining unit 140 may determine whether or not amotion artifact has been generated. For example, the motion artifactdetermining unit 140 may determine a generation of the motion artifactbased on a sum of acquired phase difference signals during one periodnot being constant (refer to equation 5 below) when the motion artifacthas been generated.

The image processor 150 may generate a depth image from which the motionartifact has been removed, and may generate a depth image from which themotion artifact and an offset have been removed, based on the depthimage from which the motion artifact has been removed and an existingdepth image from which the motion artifact has not been removed, forexample, by combining the depth image from which the motion artifact hasbeen removed and the existing depth image from which the motion artifacthas not been removed.

FIG. 2 is a diagram showing an irradiation light signal, a reflectionlight signal, and phase difference signals according to exampleembodiments.

FIG. 2A illustrates an irradiation light signal, and the irradiationlight signal S₀ may be expressed as Equation 1 shown below.

s ₀(t)=A ₀ cos(w ₀ t)  [Equation 1]

As described above, the irradiation light signal may have variouswaveforms, however, hereinafter, the irradiation light signal has acosine wave for convenience of explanation. Here A_(o) denotes anamplitude of the irradiation light signal and w_(o) denotes a frequencyof the irradiation light signal.

FIG. 2B illustrates a reflection light signal S from an object 50, andthe reflection light signal S may be expressed as Equation 2 shownbelow.

s(t)=A cos(w ₀ t−φ)+B  [Equation 2]

Since the irradiation light signal has a cosine waveform, the reflectionlight signal may have a cosine waveform. Here A denotes an amplitude ofthe reflection light signal and w_(o) denotes a frequency of theirradiation light signal. Also, B denotes an offset signal which isgenerated by other light such as background radiation, etc. and Φdenotes a phase difference by the TOF.

The sensor 130 may sense the reflection light signals according tocontrol signals whose phases are different from one another, i.e.,control signals that are out of phase with each other. For example, thecontroller 120 may generate four control signals m0, m90, m180, and m270which have a 90° phase difference from one another. For convenience ofexplanation, the four control signals are referred to as first throughfourth control signals.

Referring to FIGS. 2C through 2F, the second control signal m90 has a90° phase difference with respect to the first control signal m0, thethird control signal m180 has a 180° phase difference with respect tothe first control signal m0, and the fourth control signal m270 has a270° phase difference with respect to the first control signal m0.

The sensor 130 may generate four phase difference signals (fourintegrated charge quantities) by sensing the reflection light signalsstarting from at 0°, 90°, 180°, and 270° phases respectively for a halfperiod (T/2) of the reflection light signal, according to the firstthrough the fourth control signals.

For example, the sensor 130 may sense the reflection light signal viathe first control signal m0 and integrate the generated electrons to afirst integrator, sense the reflection light signal via the secondcontrol signal m90 and integrate the generated electrons to a secondintegrator, sense the reflection light signal via the third controlsignal m180 and integrate the generated electrons to a third integrator,and sense the reflection light signal via the fourth control signal m270and integrate the generated electrons to a fourth integrator.

The sensor 130 may generate a first phase difference signal Q_(0°)corresponding to the charge quantity integrated by the first integrator,a second phase difference signal Q_(90°) corresponding to the chargequantity integrated by the second integrator, a third phase differencesignal Q_(180°) corresponding to the charge quantity integrated by thethird integrator, and a fourth phase difference signal Q_(270°)corresponding to the charge quantity integrated by the fourthintegrator.

The first integrator and the third integrator may be the sameintegrator, and the second integrator and the fourth integrator may bethe same integrator. The present disclosure is not limited thereto, andthe first through fourth integrators may be composed of various type ofintegrators.

FIG. 3B shows phase difference images according to example embodiments,and FIG. 3B shows a depth image according to example embodiments. Theimage processor 150 of FIG. 1 may generate a plurality of phasedifference images based on the phase difference signals (chargequantities of the integrated electrons) generated by the sensor 130.

For example, as shown in FIG. 3A, the image processor 150 may generate afirst phase difference image 310 based on the first phase differencesignal Q_(0°), a second phase difference image 320 based on the secondphase difference signal Q_(90°), a third phase difference image 330based on the third phase difference signal Q_(180°), and a fourth phasedifference image 340 based on the fourth phase difference signalQ_(270°).

The image processor 150 may generate a depth image as shown in FIG. 3B,based on the plurality of phase difference images (the first throughfourth phase difference images).

The image processor 150 may calculate a phase difference based on theplurality of phase difference signals, and calculate a depth (adistance) of the object by using the calculated phase difference. Forexample, the image processor 150 may calculate a phase difference Φbetween the irradiation light signal and the reflection light signalusing Equation 3 shown below, based on the plurality of phase differencesignals (the integrated charge quantities). Also, a depth D of theobject 50 may be calculated using Equation 4 shown below, by using thecalculated phase difference Φ.

$\begin{matrix}{\varphi = {{arc}\; {\tan \left( \frac{Q_{270{^\circ}} - Q_{90{^\circ}}}{Q_{180{^\circ}} - Q_{0{^\circ}}} \right)}}} & \left\lbrack {{Equation}\mspace{14mu} 3} \right\rbrack \\{D = {\frac{c}{2} \cdot \frac{\Delta \; \varphi}{2\pi \; f}}} & \left\lbrack {{Equation}\mspace{14mu} 4} \right\rbrack\end{matrix}$

As described above, when the depth of the object 50 has been calculated,the image processor 150 may generate a depth image by using thecalculated depth. For example, as the depth increases, the imageprocessor 150 may make a brightness value of a pixel larger, and as thedepth decreases, the image processor 150 may make a brightness value ofa pixel smaller. On the contrary, as the depth decreases, the imageprocessor 150 may make a brightness value of a pixel larger, and as thedepth increases, the image processor 150 may make a brightness value ofa pixel smaller.

The image processor 150 may be realized by using a specified integratedcircuit (IC) or an application specific integrated circuit (ASIC).Alternatively the image processor 150 may be realized by software whichis installed in the depth image generating unit 100. In the case ofusing software, the image processor 150 may store the software in amovable storage medium such as a removable storage medium.

The image processor 150 may generate a depth image from which a motionartifact has been removed, and generate a depth image from which themotion artifact and an offset have been removed, based on the depthimage from which the motion artifact has been removed and an existingdepth image (from which the motion artifact has not been removed). Adetailed description thereof will be given below.

The motion artifact occurs when a camera (a depth image generatingapparatus) or an object moves while capturing the object. For example,when the camera or the object moves, a phase variation of a reflectedlight signal occurs. Thus, a phase difference signal (an integratedcharge quantity) changes, and the motion artifact may occur in the depthimage that is generated based on the changed phase difference signal.

Hereinafter, for the convenience of explanation, the depth imagegenerating apparatus operates using a 4-phase/2-tap method, and thefirst phase difference signal Q_(0°) and the third phase differencesignal Q_(180°) are simultaneously measured during a half period, andthen, the second phase difference signal Q90° and the fourth phasedifference signal Q270° are simultaneously measured during another halfperiod.

In a case in which the object moves when the depth image generatingapparatus 100 measures the second phase difference signal Q90° and thefourth phase difference signal Q270° with respect to an area where theobject is positioned, the second phase difference signal Q90° and thefourth phase difference signal Q270° may include a component of areflection light signal from a background as well as a component of areflection light signal from the object. In this case, if the depth ofthe object is calculated using Equations 3 and 4, an incorrectlycalculated depth value is acquired. The incorrectly calculated depthvalue is referred to as the motion artifact.

As described above, in a case in which the first phase difference Q0°and the third phase difference signal Q180° are measured during a halfperiod, and then, the second phase difference Q90° and the fourth phasedifference signal Q270° are measured during another half period, ifthere exists no offset signal B, the following Equations 5 and 6 areobtained.

Q _(0°) +Q _(180°) =Q _(270°) +Q _(90°) =K  [Equation 5]

|Q _(0°) −Q _(180°) |+|Q _(90°) −Q _(270°) |=K  [Equation 6]

The following Equations 7 and 8 may be obtained from Equations 5 and 6.

|Q _(0°) −Q _(180°) |=Q _(90°) +Q _(270°) −|Q _(90°) −Q_(270°)|  [Equation 7]

|Q _(90°) −Q _(270°) |=Q _(0°) +Q _(180°) −|Q _(0°) −Q_(180°)|  [Equation 8]

Referring to Equations 7 and 8, a value |Q_(90°)−Q_(270°)| may becalculated by using the first phase difference signal Q_(0°) and thethird phase difference signal Q_(180°), and a value |Q_(0°)-Q_(180°) maybe calculated by using the second phase difference signal Q_(90°) andthe fourth phase difference signal Q_(270°). For example, when measuringthe first phase difference signal Q_(0°) and the third phase differencesignal Q_(180°), if the object is moved, a value |Q_(0°)−Q_(180°)| maybe calculated by using the second phase difference signal Q_(90°) andthe fourth phase difference signal Q_(270°) measured when the object hasnot moved.

Meanwhile, when measuring the first phase difference signal Q_(0°) andthe third phase difference signal Q_(180°), if a sign ofQ_(0°)−Q_(180°)) is not changed despite movement of the object, and whenmeasuring the second phase difference signal Q_(90°) and the fourthphase difference signal Q_(270°), if a sign of (Q_(90°)−Q_(270°)) is notchanged despite movement of the object, then Equations 7 and 8 may beexpressed as Equations 9 and 10 shown below.

Q′ _(0°) −Q′ _(180°)=sign(Q _(0°) −Q _(180°))·(Q _(90°) +Q _(270°) −|Q_(90°) −Q _(270°)|)  [Equation 9]

Q′ _(90°) −Q′ _(270°)=sign(Q _(90°) −Q _(270°))·(Q _(0°) +Q _(180°) −|Q_(0°) −Q _(180°)|)  [Equation 10]

Q′ denotes a phase difference signal which is expected to be measuredwhen the motion artifact does not occur (that is, when the object doesnot move). That is, the measured phase difference signal is a phasedifference signal from which the motion artifact has been removed.

Equations 9 and 10 are induced in a state in which an offset signal doesnot exist. However, when the phase difference signal is measured bysensing the reflected light signal reflected from the object, the offsetsignal may be included in the phase difference signal due to other lightsuch as background radiation, etc.

In a case in which the motion artifact does not exist (that is, theobject does not move), Equation 3 for calculating the phase difference Φhas an effect of removing the offset signal which is commonly includedin the reflection light signal by performing a minus operation.Accordingly, the offset signal does not have an influence on the phasedifference (I) and the depth value D.

Meanwhile, if the offset signal B is considered, Equation 6 may beexpressed as Equation 11 shown below.

|Q _(0°) −Q _(180°) |+|Q _(90°) −Q _(270°) |=K+2B  [Equation 11]

Accordingly, Equations 9 and 10 may be expressed as Equations 12 and 13shown below.

$\begin{matrix}{{Q_{0{^\circ}}^{\prime} - Q_{180{^\circ}}^{\prime}} = {{{{sign}\left( {Q_{0{^\circ}} - Q_{180{^\circ}}} \right)} \cdot \left( {Q_{90{^\circ}} + Q_{270{^\circ}} + {2B} - {{Q_{90{^\circ}} - Q_{270{^\circ}}}}} \right)} = {{{{{sign}\left( {Q_{0{^\circ}} - Q_{180{^\circ}}} \right)}\left( {Q_{90{^\circ}} + Q_{270{^\circ}} - {{Q_{90{^\circ}} - Q_{270{^\circ}}}}} \right)} + {2{B \cdot {{sign}\left( {Q_{0{^\circ}} - Q_{180{^\circ}}} \right)}}}} = {{{{sign}\left( {Q_{0{^\circ}} - Q_{180{^\circ}}} \right)}\left( {Q_{90{^\circ}} + Q_{270{^\circ}} - {{Q_{90{^\circ}} - Q_{270{^\circ}}}}} \right)} + E_{{Blur}{({{0{^\circ}},{180{^\circ}}})}}}}}} & \left\lbrack {{Equation}\mspace{14mu} 12} \right\rbrack \\{{Q_{90{^\circ}}^{\prime} - Q_{270{^\circ}}^{\prime}} = {{{{sign}\left( {Q_{90{^\circ}} - Q_{270{^\circ}}} \right)} \cdot \left( {Q_{0{^\circ}} + Q_{180{^\circ}} + {2B} - {{Q_{0{^\circ}} - Q_{180{^\circ}}}}} \right)} = {{{{{sign}\left( {Q_{90{^\circ}} - Q_{270{^\circ}}} \right)} \cdot \left( {Q_{0{^\circ}} + Q_{180{^\circ}} - {{Q_{0{^\circ}} - Q_{180{^\circ}}}}} \right)} + {2{B \cdot {{sign}\left( {Q_{90{^\circ}} - Q_{270{^\circ}}} \right)}}}} = {{{{sign}\left( {Q_{90{^\circ}} - Q_{270{^\circ}}} \right)} \cdot \left( {Q_{0{^\circ}} + Q_{180{^\circ}} - \left. {Q_{0{^\circ}} - Q_{180{^\circ}}} \right)} \right.} + E_{{blur}{({{90{^\circ}},{180{^\circ}}})}}}}}} & \left\lbrack {{Equation}\mspace{14mu} 13} \right\rbrack\end{matrix}$

Therefore, as shown in Equations 12 and 13, a difference between thephase difference signals Q′_(0°)−Q′_(180°) or Q′_(90°)−Q_(270°), fromwhich the motion artifact has been removed, includes value (E_(Blur))due to the offset signal.

Referring to FIG. 4, FIG. 4A shows a depth image to which Equation 3 hasbeen applied (a depth image from which the motion artifact has not beenremoved), and FIG. 4B shows a depth image to which Equation 12 or 13 hasbeen applied (a depth image from which the motion artifact has beenremoved). Although the motion artifact has been removed from the depthimage in FIG. 4B, a whole depth value of the depth image of FIG. 4B hasbeen changed due to an effect of the offset signal. That is, a wholecolor has been changed.

Therefore, the depth image generating apparatus according to exampleembodiments may generate a depth image from which a motion artifact andan offset signal have been removed, based on an existing depth imagefrom which the motion artifact has not been removed (a depth image towhich Equation 3 has been applied) and a depth image from which themotion artifact has been removed (a depth image to which Equation 12 or13 has been applied).

For example, the depth image generating apparatus may acquire depthinformation by removing an offset effect from the existing depth image(hereinafter, referred to as first depth image), and may acquire shapeinformation by removing a motion artifact from the first depth image(hereinafter, referred to as second depth image). A detailed descriptionthereof will be described below.

The depth image generating apparatus 100 may determine an area where themotion artifact has occurred. For example, Equation 5 must be satisfiedfor a whole area according to TOF theory. In a case in which Equation 5has not been satisfied, this means that the motion artifact has occurredexcluding special cases such as rapid change in reflectivity of theobject and change in other light such as sunlight.

Accordingly, the motion artifact determining unit 140 may calculate anerror EMA due to the motion artifact by using Equation 14 shown below,and determine a motion artifact area based on the error EMA.

$\begin{matrix}{E_{MA} = {{Q_{0{^\circ}} + Q_{180{^\circ}} - \left( {Q_{90{^\circ}} + Q_{270{^\circ}}} \right)}}} & \left\lbrack {{Equation}\mspace{14mu} 14} \right\rbrack \\{{B(x)} = \left\{ \begin{matrix}{{{E_{MA}(x)} \geq {th}},{{motion}\mspace{14mu} {artifact}\mspace{14mu} {area}}} \\{{{E_{MA}(x)} < {th}},{{non}\text{-}{motion}\mspace{14mu} {artifact}\mspace{14mu} {area}}}\end{matrix} \right.} & \left\lbrack {{Equation}\mspace{14mu} 15} \right\rbrack\end{matrix}$

As shown in Equation 15, the motion artifact determining unit 140compares the calculated error E_(MA) of an area of the object with acritical value th, and when the error EMA is equal to or greater thanthe critical value th, the motion artifact determining unit 140determines the area as a motion artifact area, and when the error EMA isless than the critical value th, the motion artifact determining unit140 determines the area as a non-motion artifact area. The criticalvalue th is a constant which is determined according to noise level of asystem or a scene.

The image processor 150 may align depth values of an area in the firstdepth image and the second depth image, from which an area that has beendetermined as a motion artifact area, has been excluded, in an order ofamplitude.

For example, a first curve 510 of FIG. 5 shows aligned depth values(pixel values) of a remaining area of the first depth image from which amotion artifact area has been excluded and a second curve 520 of FIG. 5shows aligned depth values of a remaining area of the second depthimage, from which a motion artifact area has been excluded. The aligneddepth values of the curves 510 and 520 are aligned in the order ofamplitude thereof.

The depth image generating apparatus 100 can change depth values of thesecond depth image from which the motion artifact has been removed,based on corresponding relationships in the depth values of the firstdepth image and the second depth image.

For example, the depth image generating apparatus 100 can change firstdepth value in the second curve 520 into a corresponding second depthvalue in the first curve 510, based on the first curve 510 and thesecond curve 520. That is, a depth value of a pixel having a pixel index2 in the second curve 520 can be changed into a depth value of a pixelhaving the pixel index 2 in the first curve 510. Depth values ofremaining pixels in the second curve 520 can be changed in the samemanner as described above.

Accordingly, as shown in FIG. 4C, the depth image generating apparatus100 may generate a depth image (hereinafter referred to as fourth depthimage) from which a motion artifact has been removed and in which depthmatching has been performed.

The depth image generating apparatus 100 may generate a depth image fromwhich a motion artifact and an offset effect have been removed bycombining the first depth image and the fourth image. The depth imagegenerating apparatus 100 may calculate a weight factor W with respect tothe first depth image and the fourth depth image, based on an amplitudeof the motion artifact, by using Equation 16 shown below.

$\begin{matrix}{{W(x)} = \frac{1}{^{s\; 1*{({{E_{MA}{(x)}} - {s\; 2}})}}}} & \left\lbrack {{Equation}\mspace{14mu} 16} \right\rbrack\end{matrix}$

Where E_(MA) denotes an error value which is calculated using Equation14, 51 denotes a slope of the weight factor W value which changesaccording to E_(MA), and s2 denotes an E_(MA) value from which theweight factor W value starts to change. S1 and s2 may be values whichare determined by experiment.

As the motion artifact increases, the weight factor W gets closer tozero, and as the motion artifact decreases, the weight factor W getscloser to 1. That is, the first depth image and the fourth depth imagemay be combined by increasing the weight factor W of the fourth depthimage in a high motion artifact area, and increasing the weight factor Wof the first depth image in a low motion artifact area.

Meanwhile, the depth image generating apparatus 100 may combine thefirst depth image and the fourth depth image by using a wavelettransform. For example, the depth image generating apparatus 100 mayperform the wavelet transform with respect to the first depth image, andacquire a wavelet coefficient G₀ of the first depth image. Also, thedepth image generating apparatus 100 may perform the wavelet transformwith respect to the fourth depth image, and acquire a waveletcoefficient G_(M) of the fourth depth image.

The depth image generating apparatus 100 may calculate a waveletcoefficient G_(F) based on the wavelet coefficient G₀ of the first depthimage, the wavelet coefficient G_(M) of the fourth depth image, and theweight factor W calculated using Equation 16, by using Equation 17 shownbelow.

G _(F) =G _(M) +W·(G ₀ −G _(M))  [Equation 17]

The depth image generating apparatus 100 may generate a new depth image(hereinafter, referred to as a third depth image) as shown in FIG. 6, byperforming an inverse-wavelet transform on the wavelet coefficient G_(F)calculated by Equation 17. The third depth image has a depth value ofthe first depth image (from which an offset effect has been removed),and may be an image from which the motion artifact has been removed.That is, the third depth image may include the depth information of thefirst depth image and the shape information of the depth image.

FIG. 7 is a flowchart illustrating a method of generating a depth imageaccording to example embodiments.

Referring to FIG. 7, an irradiation light signal may be emitted onto anobject (Operation S710).

The irradiation light signal may be a NIR ray having a wavelength ofabout 850 nm, which is invisible to the naked eye, and may have acontinuous periodic function form with a predetermined period. Forexample, the irradiation light signal may have a specially definedwaveform such as a sine wave, a ramp wave, a rectangular wave, etc., andoptionally, the irradiation light signal may have a waveform which isnot generally defined (or may have an undefined waveform).

A plurality of phase difference signals which have different phasedifferences from one another, may be acquired by sensing a reflectionlight signal which is reflected from the object (Operation S720).

For example, the reflection light signal may be sensed by a plurality ofcontrol signals which have different phase differences from one another,and the plurality of control signals may include four control signalswhich have a 90° phase difference from one another. Accordingly, firstthrough fourth phase difference signals Q_(0°), Q_(90°), Q_(180°), andQ_(270°) may be acquired according to first through fourth controlsignals.

A first depth image may be generated based on the acquired plurality ofphase difference signals (Operation S730).

For example, a phase difference Φ between the irradiation light signaland the reflection light signal may be calculated by using the firstthrough the fourth phase difference signals and Equation 3. Also, adepth of the object may be calculated by using the calculated phasedifference Φ in Equation 4.

If the depth of the object is calculated, then the first depth image maybe generated by using the calculated depth. For example, as a depthincreases, a brightness value of a pixel may be controlled to be larger,and as a depth decreases, a brightness value of a pixel may becontrolled to be smaller. On the contrary, as a depth decreases, abrightness value of a pixel may be controlled to be larger, and as adepth increases, a brightness value of a pixel may be controlled to besmaller. Example embodiments are not limited thereto.

A second depth image may be generated based on the reflection lightsignals, in which a motion artifact has not occurred, among theplurality of reflection light signals (Operation S740).

For example, a depth and a phase difference Φ between the irradiationlight signal and the reflection light signal from which the motionartifact has been removed are calculated by using Equation 12 or 13, andthe second depth image is generated based thereon.

Hereinafter, a third depth image may be generated by combining the firstdepth image and the second depth image (Operation S750).

For example, an area where the motion artifact has occurred may bedetermined, and a weight factor with respect to the first depth imageand the second depth image may be calculated, based on the area wherethe motion artifact has occurred and an amplitude of the motionartifact. Accordingly, the third depth image may be generated bycombining the first depth image and the second depth image to which thecalculated weight factor has been applied.

FIG. 8 is a flowchart illustrating a method of generating a depth imageaccording to example embodiments.

Referring to FIG. 8, a first depth image may be generated (OperationS810). Also, a second depth image may be generated (Operation S820).Operation S810 corresponds to operation S730 in FIG. 7, and operationS820 corresponds to operation S740 in FIG. 7, therefore detaileddescriptions thereof will be omitted here.

Operations S830 through S850 may be an embodiment of operation S750 inFIG. 7, however, the present disclosure is not limited thereto.

A fourth depth image may be generated by matching a depth value of thesecond depth image with a depth value of the first depth image(Operation S830).

For example, the depth value of the second depth image from which amotion artifact has been removed, may be changed based on acorresponding relationship between the depth value of the first depthimage and the depth value of the second depth image. The depth value ofthe second depth image may be matched with the corresponding depth valueof the first depth image by aligning depth values (pixel values) of aremaining area of the first depth image in an amplitude order, fromwhich a motion artifact area has been excluded, and aligning depthvalues of a remaining area of the second depth image in an amplitudeorder, from which a motion artifact area has been excluded. Accordingly,the fourth depth image with respect to which depth matching has beenperformed and the depth value of the second depth image has been changedinto the depth value of the first depth image, may be generated.

A weight factor of the first depth image and the fourth depth image maybe calculated (Operation S840).

For example, the weight factor of the first depth image and the fourthdepth image may be calculated based on an amplitude of the motionartifact. As the motion artifact increases, the weight factor getscloser to zero, and as the motion artifact decreases, the weight factorgets closer to 1.

The first depth image and the fourth depth image may be combined byapplying the calculated weight factor thereto (Operation S850).

For example, the depth image and the fourth depth image may be combinedby using a wavelet transform. For example, the wavelet transform isperformed with respect to the first depth image, and a waveletcoefficient G₀ of the first depth image may be acquired. Also, thewavelet transform may be performed with respect to the fourth depthimage, and a wavelet coefficient G_(M) of the fourth depth image may beacquired.

A weight factor wavelet coefficient G_(F) may be calculated based on thewavelet coefficient G₀ of the first depth image, the wavelet coefficientG_(M) of the fourth depth image and the weight factor W calculated byusing Equation 16, and the third depth image may be generated byperforming inverse-wavelet transform on the calculated waveletcoefficient G_(F).

Accordingly, the third depth image may include the depth value of thefirst depth image, from which the offset effect has been removed, andmay be an image from which the motion artifact has been removed. Thatis, the third depth image may include depth information of the firstdepth image and shape information of the second depth image.

In addition, the depth image generating method according to exampleembodiments can also be implemented through computer readablecode/instructions recorded in/on a medium, e.g., a computer readablemedium, to control at least one processing element to implement any ofthe above described embodiments. The medium can correspond to anymedium/media permitting the storage and/or transmission of the computerreadable code.

The computer readable code can be recorded/transferred on a medium in avariety of ways, with examples of the medium including recording media,such as magnetic storage media (e.g., ROM, floppy disks, hard disks,etc.) and optical recording media (e.g., CD-ROMs, or DVDs), andtransmission media such as Internet transmission media. Thus, the mediummay be such a defined and measurable structure capable of including orcarrying a signal or information, such as a device carrying a bitstreamaccording to example embodiments. The media may also be a distributednetwork, so that the computer readable code is stored/transferred andexecuted in a distributed fashion. Furthermore, the processing elementmay include a processor or a computer processor, and processing elementsmay be distributed and/or included in a single device.

As described above, the depth image generating apparatus according toexample embodiments can generate a depth from which where a motionartifact and an offset effect have been removed by combining two depthimage.

It should be understood that example embodiments described herein shouldbe considered in a descriptive sense only and not for purposes oflimitation. Descriptions of features or aspects within each exampleembodiment should typically be considered as available for other similarfeatures in other example embodiments.

While example embodiments have been described with reference to thefigures, it will be understood by those of ordinary skill in the artthat various changes in form and details may be made therein withoutdeparting from the spirit and scope of the present concept as defined bythe following claims.

What is claimed is:
 1. A method of generating a depth image, the methodcomprising: irradiating an object with light generated from a lightsource; acquiring a plurality of phase difference signals which havedifferent phase differences from one another, by sensing a reflectionlight reflected from the object; generating a first depth image based onthe plurality of phase difference signals; generating a second depthimage based on phase difference signals in which a motion artifact hasnot occurred, among the plurality of phase difference signals; andgenerating a third depth image by combining the first depth image andthe second depth image.
 2. The method of claim 1, wherein the acquiringof the plurality of phase difference signals comprises acquiring aplurality of phase difference signals based on control signals havingdifferent phase differences from one another.
 3. The method of claim 1,wherein the generating of the first depth image comprises: generatingfirst phase difference images based on the plurality of phase differencesignals; and generating a first depth image based on the first phasedifference images, and wherein the generating of the second depth imagecomprises: generating second phase difference images based on phasedifference signals in which a motion artifact has not occurred, fromamong the plurality of phase difference signals; and generating thesecond depth image based on the second phase difference images.
 4. Themethod of claim 1, wherein the second depth image is a depth image fromwhich a motion artifact has been removed, and an offset effect isincluded in the second depth image.
 5. The method of claim 1, whereinthe generating of the third depth image comprises: generating a fourthdepth image by changing depth values of the second depth image intocorresponding depth values of the first depth image; and generating thethird depth image by combining the first depth image and the fourthdepth image.
 6. The method of claim 5, further comprising calculating aweight factor with respect to the first depth image and the fourth depthimage, based on an amplitude of the motion artifact, wherein thegenerating of the third depth image by combining the first depth imageand the fourth depth image comprises combining the first depth image andthe fourth depth image based on the calculated weight factor.
 7. Themethod of claim 5, wherein the generating of the third depth imagecomprises: acquiring a first wavelet coefficient by performing a wavelettransform on the first depth image, acquiring a second waveletcoefficient by performing the wavelet transform on the fourth depthimage, acquiring a third wavelet coefficient based on the first waveletcoefficient and the second wavelet coefficient, and acquiring a thirddepth image by performing an inverse-wavelet transform on the thirdwavelet coefficient.
 8. The method of claim 1, wherein the third depthimage comprises depth information with respect to the object shown inthe first depth image, and shape information with respect to the objectshown in the second depth image.
 9. The method of claim 1, wherein thethird depth image comprises an image from which a motion artifact shownin the first depth image and an offset effect shown in the second depthimage have been removed.
 10. A non-transitory computer-readable mediumhaving recorded thereon a program, which when executed by a computer,performs the method of claim
 1. 11. An apparatus for generating a depthimage, the apparatus comprising: a light irradiator irradiating anobject with light generated from a light source; a sensor acquiring aplurality of phase difference signals which have different phasedifferences from one another by sensing a reflection light reflectedfrom the object; and an image processor generating a first depth imagebased on the plurality of phase difference signals, generating a seconddepth image based on phase difference signals in which a motion artifacthas not occurred, from among the plurality of phase difference signals,and generating a third depth image by combining the first depth imageand the second depth image.
 12. The apparatus of claim 11, wherein thesensor acquires the plurality of phase difference signals based oncontrol signals having different phase differences from one another. 13.The apparatus of claim 11, wherein the image processor generates firstphase difference images based on the plurality of phase differencesignals, generates the first depth image based on the first phasedifference images, generates second phase difference images based onphase difference signals in which a motion artifact has not occurred,from among the plurality of phase difference signals, and generates thesecond depth image based on the second phase difference images.
 14. Theapparatus of claim 11, wherein the second depth image comprises a depthimage from which a motion artifact has been removed, and an offseteffect is included in the second depth image.
 15. The apparatus of claim11, wherein the image processor generates a fourth depth image bychanging depth values of the second depth image into corresponding depthvalues of the first depth image, and generates a third depth image bycombining the first depth image and the fourth depth image.
 16. Theapparatus of claim 15, wherein the image processor calculates a weightfactor with respect to the first depth image and the fourth depth imagebased on an amplitude of the motion artifact, and combines the firstdepth image and the fourth depth image based on the calculated weightfactor.
 17. The apparatus of claim 15, wherein the image processoracquires a first wavelet coefficient by performing a wavelet transformon the first depth image, acquires a second wavelet coefficient byperforming the wavelet transform on the fourth depth image, acquires athird wavelet coefficient based on the first wavelet coefficient and thesecond wavelet coefficient, and acquires the third depth image byperforming inverse-wavelet transform on the third wavelet coefficient.18. The apparatus of claim 11, wherein the third depth image comprisesdepth information with respect to the object shown in the first depthimage, and shape information with respect to the object shown in thesecond depth image.
 19. The method of claim 11, wherein the third depthimage comprises a depth image from which a motion artifact shown in thefirst depth image and an offset effect shown in the second depth imagehave been removed.
 20. A method of generating a depth image, the methodcomprising: acquiring a plurality of phase difference signals that areeach out of phase with one another by sensing a reflection lightreflected from an object irradiated with light; generating a first depthimage based on the plurality of phase difference signals; generating asecond depth image based on phase difference signals in which a motionartifact has not occurred, from among the plurality of phase differencesignals; and generating a third depth image by combining the first depthimage and the second depth image.